Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Segment Replication] Refactor RemoteStoreReplicationSource #8767

Merged
merged 26 commits into from
Aug 2, 2023

Conversation

dreamer-89
Copy link
Member

@dreamer-89 dreamer-89 commented Jul 19, 2023

Description

Refactors the RemoteStoreReplicationSource to align with existing node node implementation.

  1. Uses ReplicationCheckpoint taken from remote store instead of re-using replica's ReplicationCheckpoint. In order to do that, updated the RemoteSegmentMetadata to store ReplicationCheckpoint
  2. Refactors SegmentReplicationTarget to be re-used for remote store.
  3. Segregates shard level tests for node to node and remote store segment replication.
  4. Updated remote store unit test classes to correctly use RemoteStoreReplicationSource.

Request review from

  1. @sachinpkale for core changes (addition of replication checkpoint to uploaded metadata)
  2. @ankitkala for changes in existing segment replication integration with remote store
  3. @mch2 for overall segment replication workflow and unit test coverage post segregation

Related Issues

Resolves #7802 #8452

Check List

  • New functionality includes testing.
    • All tests pass
  • New functionality has been documented.
    • New functionality has javadoc added
  • Commits are signed per the DCO using --signoff
  • Commit changes are listed out in CHANGELOG.md file (See: Changelog)

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

@github-actions
Copy link
Contributor

Gradle Check (Jenkins) Run Completed with:

Copy link
Member

@ankitkala ankitkala left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Overall PR looks much better. Have few minor comments.

@dreamer-89 dreamer-89 marked this pull request as ready for review July 20, 2023 22:24
@github-actions

This comment was marked as outdated.

@github-actions

This comment was marked as outdated.

Signed-off-by: Suraj Singh <surajrider@gmail.com>
Signed-off-by: Suraj Singh <surajrider@gmail.com>
Signed-off-by: Suraj Singh <surajrider@gmail.com>
Signed-off-by: Suraj Singh <surajrider@gmail.com>
Signed-off-by: Suraj Singh <surajrider@gmail.com>
Signed-off-by: Suraj Singh <surajrider@gmail.com>
@github-actions

This comment was marked as outdated.

@github-actions

This comment was marked as outdated.

Signed-off-by: Suraj Singh <surajrider@gmail.com>
@github-actions
Copy link
Contributor

github-actions bot commented Aug 2, 2023

Gradle Check (Jenkins) Run Completed with:

@dreamer-89
Copy link
Member Author

dreamer-89 commented Aug 2, 2023

Gradle Check (Jenkins) Run Completed with:

#8329

org.mockito.exceptions.verification.TooManyActualInvocations: 
perFieldCorrelationVectorsFormat.getKnnVectorsFormatForField(
    "test_vector_one"
);
Wanted 1 time:
-> at org.opensearch.plugin.correlation.core.index.codec.correlation950.CorrelationCodecTests.testCorrelationVectorIndex(CorrelationCodecTests.java:108)
But was 2 times:
-> at org.apache.lucene.codecs.perfield.PerFieldKnnVectorsFormat$FieldsWriter.getInstance(PerFieldKnnVectorsFormat.java:133)
-> at org.apache.lucene.codecs.perfield.PerFieldKnnVectorsFormat$FieldsWriter.getInstance(PerFieldKnnVectorsFormat.java:133)

@dreamer-89
Copy link
Member Author

dreamer-89 commented Aug 2, 2023

checkCompatibility is failing, though there are no changes in core components which breaks plugins. This is possibly due to downstream plugins not in sync with main. Retrying!

@gaiksaya: I see you added this check via #8486. Can you confirm why this task is failing ?

> Task :checkCompatibility
Checking compatibility for: https://github.com/opensearch-project/security.git with ref: main
Checking compatibility for: https://github.com/opensearch-project/index-management.git with ref: main
Checking compatibility for: https://github.com/opensearch-project/geospatial.git with ref: main

/tmp/groovy-generated-tmpdir-15525992211363152092/src/main/java/org/opensearch/geospatial/plugin/GeospatialPlugin.java:25: error: package org.opensearch.common.component does not exist
import org.opensearch.common.component.LifecycleComponent;
                                      ^
/tmp/groovy-generated-tmpdir-15525992211363152092/src/main/java/org/opensearch/geospatial/plugin/GeospatialPlugin.java:126: error: cannot find symbol
    public Collection<Class<? extends LifecycleComponent>> getGuiceServiceClasses() {
                                      ^
  symbol:   class LifecycleComponent
  location: class GeospatialPlugin
/tmp/groovy-generated-tmpdir-15525992211363152092/src/main/java/org/opensearch/geospatial/ip2geo/listener/Ip2GeoListener.java:26: error: package org.opensearch.common.component does not exist
import org.opensearch.common.component.AbstractLifecycleComponent;
                                      ^
/tmp/groovy-generated-tmpdir-15525992211363152092/src/main/java/org/opensearch/geospatial/ip2geo/listener/Ip2GeoListener.java:38: error: cannot find symbol
public class Ip2GeoListener extends AbstractLifecycleComponent implements ClusterStateListener {
                                    ^
  symbol: class AbstractLifecycleComponent
4 errors

FAILURE: Build failed with an exception.

* What went wrong:
Execution failed for task ':compileJava'.
> Compilation failed; see the compiler error output for details.

@github-actions
Copy link
Contributor

github-actions bot commented Aug 2, 2023

Gradle Check (Jenkins) Run Completed with:

@opensearch-trigger-bot
Copy link
Contributor

Compatibility status:



> Task :checkCompatibility
Incompatible components: [https://github.com/opensearch-project/geospatial.git, https://github.com/opensearch-project/index-management.git, https://github.com/opensearch-project/security-analytics.git, https://github.com/opensearch-project/cross-cluster-replication.git, https://github.com/opensearch-project/anomaly-detection.git, https://github.com/opensearch-project/asynchronous-search.git, https://github.com/opensearch-project/performance-analyzer.git, https://github.com/opensearch-project/reporting.git]
Compatible components: [https://github.com/opensearch-project/security.git, https://github.com/opensearch-project/notifications.git, https://github.com/opensearch-project/neural-search.git, https://github.com/opensearch-project/job-scheduler.git, https://github.com/opensearch-project/sql.git, https://github.com/opensearch-project/opensearch-oci-object-storage.git, https://github.com/opensearch-project/observability.git, https://github.com/opensearch-project/k-nn.git, https://github.com/opensearch-project/alerting.git, https://github.com/opensearch-project/common-utils.git, https://github.com/opensearch-project/performance-analyzer-rca.git, https://github.com/opensearch-project/ml-commons.git]

BUILD SUCCESSFUL in 26m 21s

@github-actions
Copy link
Contributor

github-actions bot commented Aug 2, 2023

Gradle Check (Jenkins) Run Completed with:

  • RESULT: UNSTABLE ❕
  • TEST FAILURES:
      1 org.opensearch.backwards.MixedClusterClientYamlTestSuiteIT.test {p0=cluster.allocation_explain/10_basic/cluster shard allocation explanation test with empty request}

@dreamer-89 dreamer-89 merged commit 57eb105 into opensearch-project:main Aug 2, 2023
@dreamer-89 dreamer-89 added the backport 2.x Backport to 2.x branch label Aug 2, 2023
@opensearch-trigger-bot
Copy link
Contributor

The backport to 2.x failed:

The process '/usr/bin/git' failed with exit code 128

To backport manually, run these commands in your terminal:

# Fetch latest updates from GitHub
git fetch
# Create a new working tree
git worktree add ../.worktrees/backport-2.x 2.x
# Navigate to the new working tree
pushd ../.worktrees/backport-2.x
# Create a new branch
git switch --create backport/backport-8767-to-2.x
# Cherry-pick the merged commit of this pull request and resolve the conflicts
git cherry-pick -x --mainline 1 57eb105a6a156e3a313efd58e267d772ac178eea
# Push it to GitHub
git push --set-upstream origin backport/backport-8767-to-2.x
# Go back to the original working tree
popd
# Delete the working tree
git worktree remove ../.worktrees/backport-2.x

Then, create a pull request where the base branch is 2.x and the compare/head branch is backport/backport-8767-to-2.x.

dreamer-89 added a commit to dreamer-89/OpenSearch that referenced this pull request Aug 2, 2023
…ch-project#8767)

* [Segment Replication] Refactor remote replication source

Signed-off-by: Suraj Singh <surajrider@gmail.com>

* Unit test updates

Signed-off-by: Suraj Singh <surajrider@gmail.com>

* Self review

Signed-off-by: Suraj Singh <surajrider@gmail.com>

* Self review

Signed-off-by: Suraj Singh <surajrider@gmail.com>

* Segregate shard level tests for node to node and remote store segment replication

Signed-off-by: Suraj Singh <surajrider@gmail.com>

* Fix failing unit tests

Signed-off-by: Suraj Singh <surajrider@gmail.com>

* Fix failing UT

Signed-off-by: Suraj Singh <surajrider@gmail.com>

* Fix failing UT

Signed-off-by: Suraj Singh <surajrider@gmail.com>

* Address review comments

Signed-off-by: Suraj Singh <surajrider@gmail.com>

* Fix more unit tests

Signed-off-by: Suraj Singh <surajrider@gmail.com>

* Improve RemoteStoreReplicationSourceTests, remove unnecessary mocks and use actual failures for failure/exception use cases

Signed-off-by: Suraj Singh <surajrider@gmail.com>

* Spotless check fix

Signed-off-by: Suraj Singh <surajrider@gmail.com>

* Address review comments

Signed-off-by: Suraj Singh <surajrider@gmail.com>

* Ignore files already in store while computing segment file diff with primary

Signed-off-by: Suraj Singh <surajrider@gmail.com>

* Spotless fix

Signed-off-by: Suraj Singh <surajrider@gmail.com>

* Fix failing UT

Signed-off-by: Suraj Singh <surajrider@gmail.com>

* Spotless fix

Signed-off-by: Suraj Singh <surajrider@gmail.com>

* Move read/writes from IndexInput/Output to RemoteSegmentMetadata

Signed-off-by: Suraj Singh <surajrider@gmail.com>

* Address review commnt

Signed-off-by: Suraj Singh <surajrider@gmail.com>

* Update recovery flow to perform commits during recovery

Signed-off-by: Suraj Singh <surajrider@gmail.com>

* Remove un-necessary char

Signed-off-by: Suraj Singh <surajrider@gmail.com>

* Address review comments

Signed-off-by: Suraj Singh <surajrider@gmail.com>

* Update comment nit-pick

Signed-off-by: Suraj Singh <surajrider@gmail.com>

* Remove deletion logic causing read issues due to deleted segments_N

Signed-off-by: Suraj Singh <surajrider@gmail.com>

* Spotless fix

Signed-off-by: Suraj Singh <surajrider@gmail.com>

* Fix unit tests

Signed-off-by: Suraj Singh <surajrider@gmail.com>

---------

Signed-off-by: Suraj Singh <surajrider@gmail.com>
dreamer-89 added a commit to dreamer-89/OpenSearch that referenced this pull request Aug 2, 2023
…ch-project#8767)

* [Segment Replication] Refactor remote replication source

Signed-off-by: Suraj Singh <surajrider@gmail.com>

* Unit test updates

Signed-off-by: Suraj Singh <surajrider@gmail.com>

* Self review

Signed-off-by: Suraj Singh <surajrider@gmail.com>

* Self review

Signed-off-by: Suraj Singh <surajrider@gmail.com>

* Segregate shard level tests for node to node and remote store segment replication

Signed-off-by: Suraj Singh <surajrider@gmail.com>

* Fix failing unit tests

Signed-off-by: Suraj Singh <surajrider@gmail.com>

* Fix failing UT

Signed-off-by: Suraj Singh <surajrider@gmail.com>

* Fix failing UT

Signed-off-by: Suraj Singh <surajrider@gmail.com>

* Address review comments

Signed-off-by: Suraj Singh <surajrider@gmail.com>

* Fix more unit tests

Signed-off-by: Suraj Singh <surajrider@gmail.com>

* Improve RemoteStoreReplicationSourceTests, remove unnecessary mocks and use actual failures for failure/exception use cases

Signed-off-by: Suraj Singh <surajrider@gmail.com>

* Spotless check fix

Signed-off-by: Suraj Singh <surajrider@gmail.com>

* Address review comments

Signed-off-by: Suraj Singh <surajrider@gmail.com>

* Ignore files already in store while computing segment file diff with primary

Signed-off-by: Suraj Singh <surajrider@gmail.com>

* Spotless fix

Signed-off-by: Suraj Singh <surajrider@gmail.com>

* Fix failing UT

Signed-off-by: Suraj Singh <surajrider@gmail.com>

* Spotless fix

Signed-off-by: Suraj Singh <surajrider@gmail.com>

* Move read/writes from IndexInput/Output to RemoteSegmentMetadata

Signed-off-by: Suraj Singh <surajrider@gmail.com>

* Address review commnt

Signed-off-by: Suraj Singh <surajrider@gmail.com>

* Update recovery flow to perform commits during recovery

Signed-off-by: Suraj Singh <surajrider@gmail.com>

* Remove un-necessary char

Signed-off-by: Suraj Singh <surajrider@gmail.com>

* Address review comments

Signed-off-by: Suraj Singh <surajrider@gmail.com>

* Update comment nit-pick

Signed-off-by: Suraj Singh <surajrider@gmail.com>

* Remove deletion logic causing read issues due to deleted segments_N

Signed-off-by: Suraj Singh <surajrider@gmail.com>

* Spotless fix

Signed-off-by: Suraj Singh <surajrider@gmail.com>

* Fix unit tests

Signed-off-by: Suraj Singh <surajrider@gmail.com>

---------

Signed-off-by: Suraj Singh <surajrider@gmail.com>
dreamer-89 added a commit to dreamer-89/OpenSearch that referenced this pull request Aug 3, 2023
…ch-project#8767)

* [Segment Replication] Refactor remote replication source

Signed-off-by: Suraj Singh <surajrider@gmail.com>

* Unit test updates

Signed-off-by: Suraj Singh <surajrider@gmail.com>

* Self review

Signed-off-by: Suraj Singh <surajrider@gmail.com>

* Self review

Signed-off-by: Suraj Singh <surajrider@gmail.com>

* Segregate shard level tests for node to node and remote store segment replication

Signed-off-by: Suraj Singh <surajrider@gmail.com>

* Fix failing unit tests

Signed-off-by: Suraj Singh <surajrider@gmail.com>

* Fix failing UT

Signed-off-by: Suraj Singh <surajrider@gmail.com>

* Fix failing UT

Signed-off-by: Suraj Singh <surajrider@gmail.com>

* Address review comments

Signed-off-by: Suraj Singh <surajrider@gmail.com>

* Fix more unit tests

Signed-off-by: Suraj Singh <surajrider@gmail.com>

* Improve RemoteStoreReplicationSourceTests, remove unnecessary mocks and use actual failures for failure/exception use cases

Signed-off-by: Suraj Singh <surajrider@gmail.com>

* Spotless check fix

Signed-off-by: Suraj Singh <surajrider@gmail.com>

* Address review comments

Signed-off-by: Suraj Singh <surajrider@gmail.com>

* Ignore files already in store while computing segment file diff with primary

Signed-off-by: Suraj Singh <surajrider@gmail.com>

* Spotless fix

Signed-off-by: Suraj Singh <surajrider@gmail.com>

* Fix failing UT

Signed-off-by: Suraj Singh <surajrider@gmail.com>

* Spotless fix

Signed-off-by: Suraj Singh <surajrider@gmail.com>

* Move read/writes from IndexInput/Output to RemoteSegmentMetadata

Signed-off-by: Suraj Singh <surajrider@gmail.com>

* Address review commnt

Signed-off-by: Suraj Singh <surajrider@gmail.com>

* Update recovery flow to perform commits during recovery

Signed-off-by: Suraj Singh <surajrider@gmail.com>

* Remove un-necessary char

Signed-off-by: Suraj Singh <surajrider@gmail.com>

* Address review comments

Signed-off-by: Suraj Singh <surajrider@gmail.com>

* Update comment nit-pick

Signed-off-by: Suraj Singh <surajrider@gmail.com>

* Remove deletion logic causing read issues due to deleted segments_N

Signed-off-by: Suraj Singh <surajrider@gmail.com>

* Spotless fix

Signed-off-by: Suraj Singh <surajrider@gmail.com>

* Fix unit tests

Signed-off-by: Suraj Singh <surajrider@gmail.com>

---------

Signed-off-by: Suraj Singh <surajrider@gmail.com>
dreamer-89 added a commit that referenced this pull request Aug 3, 2023
…9076)

* [Segment Replication] Refactor RemoteStoreReplicationSource (#8767)

* [Segment Replication] Refactor remote replication source

Signed-off-by: Suraj Singh <surajrider@gmail.com>

* Unit test updates

Signed-off-by: Suraj Singh <surajrider@gmail.com>

* Self review

Signed-off-by: Suraj Singh <surajrider@gmail.com>

* Self review

Signed-off-by: Suraj Singh <surajrider@gmail.com>

* Segregate shard level tests for node to node and remote store segment replication

Signed-off-by: Suraj Singh <surajrider@gmail.com>

* Fix failing unit tests

Signed-off-by: Suraj Singh <surajrider@gmail.com>

* Fix failing UT

Signed-off-by: Suraj Singh <surajrider@gmail.com>

* Fix failing UT

Signed-off-by: Suraj Singh <surajrider@gmail.com>

* Address review comments

Signed-off-by: Suraj Singh <surajrider@gmail.com>

* Fix more unit tests

Signed-off-by: Suraj Singh <surajrider@gmail.com>

* Improve RemoteStoreReplicationSourceTests, remove unnecessary mocks and use actual failures for failure/exception use cases

Signed-off-by: Suraj Singh <surajrider@gmail.com>

* Spotless check fix

Signed-off-by: Suraj Singh <surajrider@gmail.com>

* Address review comments

Signed-off-by: Suraj Singh <surajrider@gmail.com>

* Ignore files already in store while computing segment file diff with primary

Signed-off-by: Suraj Singh <surajrider@gmail.com>

* Spotless fix

Signed-off-by: Suraj Singh <surajrider@gmail.com>

* Fix failing UT

Signed-off-by: Suraj Singh <surajrider@gmail.com>

* Spotless fix

Signed-off-by: Suraj Singh <surajrider@gmail.com>

* Move read/writes from IndexInput/Output to RemoteSegmentMetadata

Signed-off-by: Suraj Singh <surajrider@gmail.com>

* Address review commnt

Signed-off-by: Suraj Singh <surajrider@gmail.com>

* Update recovery flow to perform commits during recovery

Signed-off-by: Suraj Singh <surajrider@gmail.com>

* Remove un-necessary char

Signed-off-by: Suraj Singh <surajrider@gmail.com>

* Address review comments

Signed-off-by: Suraj Singh <surajrider@gmail.com>

* Update comment nit-pick

Signed-off-by: Suraj Singh <surajrider@gmail.com>

* Remove deletion logic causing read issues due to deleted segments_N

Signed-off-by: Suraj Singh <surajrider@gmail.com>

* Spotless fix

Signed-off-by: Suraj Singh <surajrider@gmail.com>

* Fix unit tests

Signed-off-by: Suraj Singh <surajrider@gmail.com>

---------

Signed-off-by: Suraj Singh <surajrider@gmail.com>

* Rebase 2.x and resolve conflicts

Signed-off-by: Suraj Singh <surajrider@gmail.com>

* Fix unit test

Signed-off-by: Suraj Singh <surajrider@gmail.com>

---------

Signed-off-by: Suraj Singh <surajrider@gmail.com>
kaushalmahi12 pushed a commit to kaushalmahi12/OpenSearch that referenced this pull request Sep 12, 2023
…ch-project#8767)

* [Segment Replication] Refactor remote replication source

Signed-off-by: Suraj Singh <surajrider@gmail.com>

* Unit test updates

Signed-off-by: Suraj Singh <surajrider@gmail.com>

* Self review

Signed-off-by: Suraj Singh <surajrider@gmail.com>

* Self review

Signed-off-by: Suraj Singh <surajrider@gmail.com>

* Segregate shard level tests for node to node and remote store segment replication

Signed-off-by: Suraj Singh <surajrider@gmail.com>

* Fix failing unit tests

Signed-off-by: Suraj Singh <surajrider@gmail.com>

* Fix failing UT

Signed-off-by: Suraj Singh <surajrider@gmail.com>

* Fix failing UT

Signed-off-by: Suraj Singh <surajrider@gmail.com>

* Address review comments

Signed-off-by: Suraj Singh <surajrider@gmail.com>

* Fix more unit tests

Signed-off-by: Suraj Singh <surajrider@gmail.com>

* Improve RemoteStoreReplicationSourceTests, remove unnecessary mocks and use actual failures for failure/exception use cases

Signed-off-by: Suraj Singh <surajrider@gmail.com>

* Spotless check fix

Signed-off-by: Suraj Singh <surajrider@gmail.com>

* Address review comments

Signed-off-by: Suraj Singh <surajrider@gmail.com>

* Ignore files already in store while computing segment file diff with primary

Signed-off-by: Suraj Singh <surajrider@gmail.com>

* Spotless fix

Signed-off-by: Suraj Singh <surajrider@gmail.com>

* Fix failing UT

Signed-off-by: Suraj Singh <surajrider@gmail.com>

* Spotless fix

Signed-off-by: Suraj Singh <surajrider@gmail.com>

* Move read/writes from IndexInput/Output to RemoteSegmentMetadata

Signed-off-by: Suraj Singh <surajrider@gmail.com>

* Address review commnt

Signed-off-by: Suraj Singh <surajrider@gmail.com>

* Update recovery flow to perform commits during recovery

Signed-off-by: Suraj Singh <surajrider@gmail.com>

* Remove un-necessary char

Signed-off-by: Suraj Singh <surajrider@gmail.com>

* Address review comments

Signed-off-by: Suraj Singh <surajrider@gmail.com>

* Update comment nit-pick

Signed-off-by: Suraj Singh <surajrider@gmail.com>

* Remove deletion logic causing read issues due to deleted segments_N

Signed-off-by: Suraj Singh <surajrider@gmail.com>

* Spotless fix

Signed-off-by: Suraj Singh <surajrider@gmail.com>

* Fix unit tests

Signed-off-by: Suraj Singh <surajrider@gmail.com>

---------

Signed-off-by: Suraj Singh <surajrider@gmail.com>
Signed-off-by: Kaushal Kumar <ravi.kaushal97@gmail.com>
brusic pushed a commit to brusic/OpenSearch that referenced this pull request Sep 25, 2023
…ch-project#8767)

* [Segment Replication] Refactor remote replication source

Signed-off-by: Suraj Singh <surajrider@gmail.com>

* Unit test updates

Signed-off-by: Suraj Singh <surajrider@gmail.com>

* Self review

Signed-off-by: Suraj Singh <surajrider@gmail.com>

* Self review

Signed-off-by: Suraj Singh <surajrider@gmail.com>

* Segregate shard level tests for node to node and remote store segment replication

Signed-off-by: Suraj Singh <surajrider@gmail.com>

* Fix failing unit tests

Signed-off-by: Suraj Singh <surajrider@gmail.com>

* Fix failing UT

Signed-off-by: Suraj Singh <surajrider@gmail.com>

* Fix failing UT

Signed-off-by: Suraj Singh <surajrider@gmail.com>

* Address review comments

Signed-off-by: Suraj Singh <surajrider@gmail.com>

* Fix more unit tests

Signed-off-by: Suraj Singh <surajrider@gmail.com>

* Improve RemoteStoreReplicationSourceTests, remove unnecessary mocks and use actual failures for failure/exception use cases

Signed-off-by: Suraj Singh <surajrider@gmail.com>

* Spotless check fix

Signed-off-by: Suraj Singh <surajrider@gmail.com>

* Address review comments

Signed-off-by: Suraj Singh <surajrider@gmail.com>

* Ignore files already in store while computing segment file diff with primary

Signed-off-by: Suraj Singh <surajrider@gmail.com>

* Spotless fix

Signed-off-by: Suraj Singh <surajrider@gmail.com>

* Fix failing UT

Signed-off-by: Suraj Singh <surajrider@gmail.com>

* Spotless fix

Signed-off-by: Suraj Singh <surajrider@gmail.com>

* Move read/writes from IndexInput/Output to RemoteSegmentMetadata

Signed-off-by: Suraj Singh <surajrider@gmail.com>

* Address review commnt

Signed-off-by: Suraj Singh <surajrider@gmail.com>

* Update recovery flow to perform commits during recovery

Signed-off-by: Suraj Singh <surajrider@gmail.com>

* Remove un-necessary char

Signed-off-by: Suraj Singh <surajrider@gmail.com>

* Address review comments

Signed-off-by: Suraj Singh <surajrider@gmail.com>

* Update comment nit-pick

Signed-off-by: Suraj Singh <surajrider@gmail.com>

* Remove deletion logic causing read issues due to deleted segments_N

Signed-off-by: Suraj Singh <surajrider@gmail.com>

* Spotless fix

Signed-off-by: Suraj Singh <surajrider@gmail.com>

* Fix unit tests

Signed-off-by: Suraj Singh <surajrider@gmail.com>

---------

Signed-off-by: Suraj Singh <surajrider@gmail.com>
Signed-off-by: Ivan Brusic <ivan.brusic@flocksafety.com>
shiv0408 pushed a commit to Gaurav614/OpenSearch that referenced this pull request Apr 25, 2024
…ch-project#8767)

* [Segment Replication] Refactor remote replication source

Signed-off-by: Suraj Singh <surajrider@gmail.com>

* Unit test updates

Signed-off-by: Suraj Singh <surajrider@gmail.com>

* Self review

Signed-off-by: Suraj Singh <surajrider@gmail.com>

* Self review

Signed-off-by: Suraj Singh <surajrider@gmail.com>

* Segregate shard level tests for node to node and remote store segment replication

Signed-off-by: Suraj Singh <surajrider@gmail.com>

* Fix failing unit tests

Signed-off-by: Suraj Singh <surajrider@gmail.com>

* Fix failing UT

Signed-off-by: Suraj Singh <surajrider@gmail.com>

* Fix failing UT

Signed-off-by: Suraj Singh <surajrider@gmail.com>

* Address review comments

Signed-off-by: Suraj Singh <surajrider@gmail.com>

* Fix more unit tests

Signed-off-by: Suraj Singh <surajrider@gmail.com>

* Improve RemoteStoreReplicationSourceTests, remove unnecessary mocks and use actual failures for failure/exception use cases

Signed-off-by: Suraj Singh <surajrider@gmail.com>

* Spotless check fix

Signed-off-by: Suraj Singh <surajrider@gmail.com>

* Address review comments

Signed-off-by: Suraj Singh <surajrider@gmail.com>

* Ignore files already in store while computing segment file diff with primary

Signed-off-by: Suraj Singh <surajrider@gmail.com>

* Spotless fix

Signed-off-by: Suraj Singh <surajrider@gmail.com>

* Fix failing UT

Signed-off-by: Suraj Singh <surajrider@gmail.com>

* Spotless fix

Signed-off-by: Suraj Singh <surajrider@gmail.com>

* Move read/writes from IndexInput/Output to RemoteSegmentMetadata

Signed-off-by: Suraj Singh <surajrider@gmail.com>

* Address review commnt

Signed-off-by: Suraj Singh <surajrider@gmail.com>

* Update recovery flow to perform commits during recovery

Signed-off-by: Suraj Singh <surajrider@gmail.com>

* Remove un-necessary char

Signed-off-by: Suraj Singh <surajrider@gmail.com>

* Address review comments

Signed-off-by: Suraj Singh <surajrider@gmail.com>

* Update comment nit-pick

Signed-off-by: Suraj Singh <surajrider@gmail.com>

* Remove deletion logic causing read issues due to deleted segments_N

Signed-off-by: Suraj Singh <surajrider@gmail.com>

* Spotless fix

Signed-off-by: Suraj Singh <surajrider@gmail.com>

* Fix unit tests

Signed-off-by: Suraj Singh <surajrider@gmail.com>

---------

Signed-off-by: Suraj Singh <surajrider@gmail.com>
Signed-off-by: Shivansh Arora <hishiv@amazon.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
backport 2.x Backport to 2.x branch skip-changelog
Projects
None yet
4 participants